Methods and apparatus for image processing and analysis

ABSTRACT

Methods and apparatus to providing image fusion using a number of image processing approaches. In one embodiment, a single image is processed and image fusion is performed. In exemplary embodiments of the invention, image enhancement, denoising, edge detection, etc., can be provided.

BACKGROUND

As is known in the art, a variety of techniques are known for enhancingimages of scenes which are obscured by backscattered light. For example,there are many known methods for enhancing the contrast of images insuch circumstances, but the maximum improvement in the quality of theimage is limited by a number of factors, as disclosed in U.S. Pat. No.6,462,768, which is incorporated herein by reference. For example, thegain of the camera or other sensing system is set, usually by anautomatic gain control, to the maximum brightness of the image. When thescattered light component is large, the transmitted terrain componentbecomes small in comparison with the quantization noise of the sensor.In addition, the backscattered light often has a random component thatis a source of noise which is amplified by any contrast-stretchingtransformation implemented by the sensor. Further, in low lightconditions, statistical fluctuations in the transmitted photon flux giverise to noise in the image. This noise will be amplified by anytransformation that increases the range of contrasts present in theimage.

Various contrast enhancement algorithms are known, for example, variancenormalization or histogram equalization, see, e.g., U.S. Pat. Nos.6,462,768, 6,982,764, 8,331,711, 5,681,112, 5,218,649, 6,876,777,5,300,169, and 6,064,775, and U.S. Patent Publications No. 2012/0275721,all of which are incorporated herein by reference. In practice, however,such known contrast enhancement algorithms have not providedparticularly good results.

The aim of image enhancement is to modify input images in such a waythat the visual content contained in the image is improved with respectto a set of defined criteria. As there is no single set of criteriawhich can universally define an ideal enhancement, many imageenhancement techniques have been proposed. The most basic of imageenhancement approaches include pixel transformations such as logarithmictransformations, gamma transformations, and contrast stretchingoperations, which define a fixed or parametrically adjustable one-to-onemapping by which the intensity values of individual pixels are modified.Histogram equalization is an automated enhancement process which usesthe histogram of the input image itself to determine the one-to-onemapping of intensity values for which an approximately uniformdistribution is yielded in the enhanced result. This procedure has beenfurther generalized to histogram matching, whereby the input histogramis matched to any defined histogram distribution. As these methods useglobal image properties to determine pixel transformations and apply thesame transformation to each pixel in the same way regardless of localimage information, they may not be appropriately applied in a localcontext and often times yield inadequate detail preservation orover-enhancement. Consequently, adaptive procedures, such ascontrast-limited adaptive histogram equalization, have been developed tolocally adapt the enhancement process based on local image features.Moreover, algorithms such as multi-scale retinex attempt to model thetransfer functions of the human optical nerve, cortex, and so forth, andformulate enhancement algorithms by implementing filters which recreatethese processes to model human vision. However, the way in which theseapproaches actually enhance, and in particular, image edges, is stillunpredictable. In this sense, the approaches may be classified asindirect image enhancement algorithms, as they enhance images andgenerally improve image contrast without explicitly defining imagecontrast itself. Conversely, direct image enhancement algorithmsquantitatively define a contrast measure in either a spatial ortransform domain, and achieve image enhancement by increasing themeasured contrast. Accordingly, direct image enhancement algorithms havebeen developed using contrast measures defined in the DCT, pyramidal,and wavelet transform domains. These algorithms are capable of enhancingfine local edge structures, but generally are less successful inimproving global image contrasts adequately even when scale parametersare chosen appropriately. Overall, it is still observed that no singleimage enhancement algorithm is capable of delivering an idealenhancement for all circumstances and practical applications.

The goal of image denoising is to remove the noise which has corruptedan image. There may be many sources of the noise itself, including theimaging devices, particularly when image signals are weak, or a noisytransmission channel. Of particular interest is the problem of removingadditive white Gaussian noise from images. The basic tradeoff whichexists in denoising is between the ability to effectively remove noisewhile also accurately preserving edges. The most basic means of Gaussiannoise denoising is Gaussian filtering. However, this approach is veryprone to blurring edges and fine details as it filters isotropically.Partial differential equation (PDE) based approaches such as anisotropicdiffusion generalize the replace of the isotropic filter with aconduction function which smoothes the image more in non-edge regionsless on edges. Total variational approaches formulate the denoisingproblem as a constrained optimization problem. Wavelet-based denoisingapproaches have also been proposed based on several means ofthresholding wavelet coefficients. Despite the formulations ofalgorithms, there will always inevitably be some tradeoff betweensufficient noise removing and accurate edge preservation.

SUMMARY

Exemplary embodiments of the invention provide methods and apparatus fora fusion-based multimedia processing system and method that can solvereal life multimedia related issues. Exemplary embodiments of theinvention provide systems and methods utilizing image fusion to performimage processing by combining advantages of different image processingapproaches. In general, embodiments of the invention improve overconventional approaches by the way in which image fusion, and thecombination of multiple images, are integrated to improve existing imageprocessing. Currently, image fusion has only been used when multiplesource images from different capture techniques or imaging modalitiesare available. Multiple source images may not be available for manypractical applications, and thus, there has been no means of utilizingimage fusion in such cases. Exemplary embodiments of the inventionprovide systematically use image fusion when only a single image isavailable.

In embodiments, the system is also capable of processing and fusingmultiple source images if they are available. The resulting system maybe used for many different image applications, such as imageenhancement, image denoising, edge detection, image resizing, imageresolution, image encryption, image standardization/coloring, andothers.

Exemplary embodiments of the invention illustrate possible uses of thesystem in the context of image enhancement and image denoising. As onlya single image is used as input to image enhancement processing, thesystem in this context enhances the image using several different imageenhancement processes, and fuses the results to obtain an enhancedresult that leverages the benefits of each of the secondaryenhancements.

In addition, since different image enhancement processes are based ondifferent criteria, exemplary embodiments of the invention provideprocessing based upon the characteristics of the different enhancementapproaches to effectively combine their advantages. In the context ofimage denoising, for example, it is generally expected that only asingle image is available as input.

One embodiment provides iterative fusion-based denoising which fusessecondary denoising results with various degrees of smoothing. A smalldegree of smoothing is insufficient for removing noise in the image, butpreserves the edges of the images. Conversely, a large degree ofsmoothing effectively removes noise but also blurs edges. Thus, thecombination of these secondary outputs by the proposed iterative schemeallows for noise to be effectively removed while also retaining imageedges.

In one aspect of the invention, a method for performing image fusion forimage processing, comprises: receiving a first input image (I);generating, using a computer processor, N secondary outputs (I_(n), n=1,2, . . . N−1, N) using N combinations of secondary image processesand/or parameter sets derived from the first input image (I); and fusingthe intermediate outputs to yield a processed output (I′).

In another aspect of the invention, a method for decoupling local andglobal contrast enhancement processes for an original image, comprises:performing, using a computer processor, indirect contrast enhancement ofthe original image to yield a globally contrast-enhanced output;decomposing the original image and the globally contrast-enhanced outputusing a multi-resolution decomposition process; selecting approximationinformation of the globally contrast-enhanced output; calculating detailinformation to restore contrast of the original image based on theapproximation information of the globally contrast-enhanced output;composing a fused output using the selected approximation and the detailinformation; and performing direct contrast enhancement to yield a finaloutput.

In a further aspect of the invention, an iterative method foredge-preserving noise removal of an image, comprises: (a) processing,using a computer processor, the image by generating secondary denoisingoutputs using varying degrees of smoothing; (b) measuring features ofthe image; (c) fusing a least aggressively smoothed image with a mostaggressively smoothed image based on the features of the image toprovide a fused output; (d) fusing the fused output with the mostaggressively smoothed image based on the features of the image; and (e)iterating steps (a)-(d) K times, replacing the more aggressivelysmoothed image with the one calculated in step (d).

In another aspect of the invention, a method of calculating HVS-basedmulti-scale transforms based on luminance and contrast maskingcharacteristics of the HVS, comprises: calculating a multi-resolutiontransform to yield a set of approximation and detail coefficients; usingthe approximation coefficients to mask the detail coefficients to yieldmulti-scale luminance-masked contrast coefficients; using theluminance-masked contrast coefficients at different levels ofdecomposition to mask the luminance-masked contrast coefficients toyield the multi-scale luminance and contrast-masked coefficients.

In another aspect of the invention, a method for multi-scale de-noisingusing HVS-based multi-scale transforms, comprises: performing, using acomputer processor, a multi-resolution decomposition of an image;de-noising the image by: de-noising contrast coefficients; and/orsmoothing approximation coefficients; inferring the smoothed contrastcoefficients using the smoothed approximation coefficients; andperforming an inverse transform to generate an output image.

In another aspect of the invention, a method for generating a set ofimage fusion processes based on an adaptive-weighting scheme based onstructural similarity, comprises: performing, using a computerprocessor, multi-resolution processing to source images; fusingapproximation coefficients to yield an initial fusion estimate of theapproximation coefficients; recalculating weights for averaging of theapproximation coefficients based on a similarity between the initialfusion estimate and each of the source images; fusing contrastcoefficients using a contrast coefficient fusion process; and performingan inverse transform to generate an output.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of this invention, as well as the inventionitself, may be more fully understood from the following description ofthe drawings in which:

FIG. 1 is a block diagram of a system having image processing fusionsystem in accordance with an embodiment of the present invention;

FIG. 2 is a diagram of a system having a plurality of processing modulesto receive and process an input image and a fusion module;

FIG. 3 is a diagram of a system having image enhancement modules toreceive and process an input image and a fusion module;

FIG. 4 is a diagram of a system having segmentation and fusion ofsub-images that are unionized;

FIG. 5 is a diagram of a system having decomposition, processingmodules, and fusion;

FIG. 6 is a diagram of decoupling of global and local image enhancement;

FIGS. 7(a) and (b) show source images and 7(c)-(h) show processedimages;

FIG. 8 is a diagram of enhancement of color images;

FIG. 9 shows a graphical representation of (a) approximation, (b)detail, and (c) HVS-based contrast coefficient sub-bands;

FIG. 10 is a diagram showing HVS image enhancement;

FIG. 11 is a diagram of image denoising;

FIG. 12 is a graphical representation of (a) image fusion, (b) imagede-noising, and (c) JFD visualized as averaging;

FIG. 13 is a diagram of (a) SFD processing and (b) JFD processing;

FIG. 14 shows (a) original image, (b) histogram equalization processedimage, (c) contrast-limited adaptive histogram equalization, and (d)image after fusion of (b) and (c); and

FIG. 15 is a representation of an exemplary computer that can perform atleast a portion of the processing described herein.

DETAILED DESCRIPTION

FIG. 1 shows an exemplary image processing system 100 having fusion inaccordance with exemplary embodiments of the invention. First, second,and third input data 102 a-M are input to each of first, second, andthird data processing modules 104 a-M. It is understood that anypractical number of input data and processing modules can be used. Theoutput of each of the data processing modules 104 a-M is provided toeach of first, second and third fusion modules 106 a-L each generating arespective output data 108 a-L. The output data 108 are provided to eachdata processing module 110 a-L the outputs of which are provided tofusion modules 112 a-L. Data processing modules 114 a-L receive theoutput from fusion modules 112 to generate respective output data-K, asdescribed more fully below.

FIG. 2 shows exemplary image enhancement of a scene. An input image 200is provided to a series of image processing modules 202 a-N eachproviding particular image enhancement processing with various parametersets to generate N secondary outputs I_(n), n=1, 2, . . . , N−1, N) fromthe single input image 200. In a sense, these secondary outputs emulatethe multiple source images which are assumed to be available inmulti-sensor data fusion applications. The secondary outputs I_(1-N) arethen provided to a fusion module 204 that yields an enhanced outputimage I′.

Due to the fusion processing, the enhanced output image I′ is aneffective combination of the secondary outputs which fuse the advantagesof each of the secondary enhancement processes. In this context, avariety of suitable image enhancement processes can be used. The outputof the system will be especially dissimilar from the secondary outputswhen the secondary outputs are themselves dissimilar.

It is understood that where image process K, k=1, 2, . . . N, can be thecommonly used or new development image enhancement processes, it can beshown that the system is a generalized and theoretically sound frameworkfor image processing. For example, when all the secondary outputs areequivalent, e.g., I₀=I₁= . . . =I_(NA)=I_(N), then I′=I₀. Therefore, anyimage enhancement process can be viewed as an instance of the inventivesystem when the same image enhancement algorithm is used to generateeach of the N secondary outputs. Furthermore, if I₀=I₁= . . .=I_(N−1)=I_(N)=I, then I′=I. Therefore, if the secondary processes areparametric for which a given parameter set yields the original image,there is a parameter set which will yield the original image by aninventive system.

Exemplary embodiments exhibit a recursive structure if the secondaryenhancements are the proposed enhancement system itself. In this case,different fusion approaches can be used in the secondary enhancementstages, as well as in the final stage, which fuses secondary outputs.Similar formulations can be derived for other image processingapplications, including the denoising application described herein. Theproposed framework in the context of image enhancement, image denoising,and edge detection are illustrated in FIGS. 3, 4, and 5, respectively.

FIG. 3 shows an image enhancement system 300 having a first enhancementmodule 302 a outputting I₁, a second enhancement module 302 b outputtingI₂, a third enhancement module 302 c outputting I_(N−1), and a fourthenhancement module 302N outputting I_(N). In other embodiments, insteadof enhancement modules, denoising modules and/or edge detection modulescan be used for image denoising/edge detection applications. It isunderstood that any practical number and type of enhancement process canbe used.

FIG. 4 shows an exemplary system 400 decomposing secondary images into Msub-images using a segmentation approach, and fusing each sub-image ofeach the secondary images. A first second image I₁ is provided to afirst segmentation module 402, which outputs sub-images I_(1,1),I_(1,2), . . . I_(1,M-1), I_(1,M), and a second image I_(N) is providedto a second segmentation module 404, which outputs sub-images I_(N,1),I_(N,2), . . . I_(N,M-1), I_(N,M). It is understood that any practicalnumber of input images and segmentation modules can be used. Thesub-images are processed by fusion modules 406 which provide outputs tounionization module 408.

It is understood that the sub-images can be fused in various ways. Forexample, Human Visual System (HVS)-based decomposition may be used todecompose sub-images based on the human eye's ability to discriminatebetween important and unimportant data. Each sub-image may then be fusedaccording to a fusion scheme which accurately addresses the features ofeach sub-image. The fused sub-images are unionized to yield the finalfused output. The fusion 406 of FIG. 4 can be used as the fusion in thegeneralized system in FIG. 1.

In one embodiment, the input image is decomposed into N grayscalerepresentations by bi-dimensional empirical mode decomposition, forexample. This differs from the system of FIG. 4 as the domains of eachrepresentation are equal, whereas the system in FIG. 5 decomposes theimage into non-overlapping regions. Each grayscale representation fromthe decomposition is separately processed and fused to yield the outputimage, as shown in FIG. 5.

In another aspect of the invention, a system effectively combinesindirect and direct image enhancement procedures. Indirect and directimage enhancement processing are based on different criteria so thattheir respective outputs may be complementary in nature. Indirectapproaches generally improve global contrast, but indirectly alter theedge information of images in an erratic manner. Consequently, theoriginal edge information of the image may either not be accuratelypreserved or may be over-enhanced. Conversely, direct image enhancementprocedures are more suitable for local edge enhancement.

In one embodiment, a method uses a priori knowledge to decouple globaland local enhancement procedures, allowing them to be tunedindependently. FIG. 6 shows an exemplary process 600 in which an imageis first subjected to a modified global contrast enhancement process,which can be viewed as a specific instance of the generalized system,where the secondary outputs I₁ and I₂ are the original image and theoutput of an indirect image enhancement process 602, respectively, andwhere the fusion process 604 makes direct use of the a priori knowledge.The direct image enhancement 606 outputs the enhanced image.

The fusion process begins by performing an L-level decomposition of I₁and I₂ using a multi-resolution decomposition scheme. This decompositiongenerates the approximation coefficient sub-bands y_(I) ₁ _(,0) ^((l)),y_(I) ₂ _(,0) ^((l)) and i detail coefficient sub-bands y_(I) ₁ _(,i)^((l)), y_(I) ₂ _(,i) ^((l)) at each decomposition level l for I₁ andI₂, respectively. For this stage, any multi-resolution decompositionscheme may be used. Here, the Laplacian Pyramid (LP) is used toillustrate the approach, in which case, i=1. Assuming that theapproximation coefficients y_(I) ₂ _(,0) ^((L)) adequately represent thedesired global contrast enhancement, the approximation coefficients ofthe fused image at level L are given byy _(I) _(enh) _(,0) ^((L)) =y _(I) ₂ _(,0) ^((L))  (1)

The detail coefficients of the fused output restore the contrastinformation of the original image in order to accurately preserve itsedges. Since the human visual system is sensitive to relative luminancechanges, the detail coefficients of the original image are scaledaccording to the luminance changes resulting from the global contrastenhancement. The exact means by which this is accomplished depends onthe multi-resolution decomposition scheme used by the fusion approach.For the case of the LP, the contrast of the original image is restoredby

$\begin{matrix}{y_{I^{\prime},i}^{(l)} = {y_{I_{1},i}^{(l)}\frac{{EXPAND}( y_{I_{2},{i + 1}}^{(l)} )}{{EXPAND}( y_{I_{1},{i + 1}}^{(l)} )}}} & (2)\end{matrix}$

The inverse transform is performed to yield the image I′. This proceduresuccessfully obtains the global contrast enhancement of the indirectenhancement algorithm while retaining the contrast of the originalimage. Applying a direct image enhancement 606 to I′ thereafter canenhance fine details while still obtaining the advantages of the globalcontrast enhancement, yielding the final enhanced output I_(enh). Thus,the processing decouples the local and global enhancement procedure fromeach other and allows them to be tuned independently.

It is understood that a known class of image fusion processing adoptsthe Parameterized Logarithmic Image Processing (PLIP) model, which is anonlinear image processing framework whose mathematical operators moreconsistently correspond to human visual system characteristics. In oneembodiment, a fusion system employs PLIP mathematical operators that maybe used in the spatial or transform domain based on visual andcomputational requirements. PLIP is described, for example, in S.Nercessian, K. Panetta, and S. Agaian, “Multiresolution DecompositionSchemes Using the Parameterized Logarithmic Image Processing Model withApplication to Image Fusion,” EURASIP Journal on Advances in SignalProcessing, vol. 2011, Article ID 515084, 17 pages, 2011.doi:10.1155/2011/515084, which is incorporated herein by reference.

The PLIP model interprets images as absorption filters known asgraytones based on the maximum range of the image M, and processes thesegraytones using a new arithmetic which replaces standard arithmeticaloperators. The resulting set of arithmetic operators can be used toprocess images based on a physically relevant image formation model. Themodel makes use of a logarithmic isomorphic transformation, consistentwith the fact that the human visual system processes lightlogarithmically. The model has also shown to satisfy Weber's Law, whichquantifies the human eye's ability to perceive intensity differences fora given background intensity. It has been shown that psychophysical lawscan be context-dependent, and thus, the constants governing thesepsychophysical laws are indeed parametric. Thus, the parametric natureof the model allows mathematical operators to be tuned according toimage-dependent characteristics. At its core, the model generalizes theisomorphic transformation originally formulated in by the LogarithmicImage Processing (LIP) model. Consequently, a new set of PLIPmathematical operators, namely addition, subtraction, and scalarmultiplication, are defined for graytones g₁ and g₂ and scalar constantc in terms of this isomorphic transformation, thus replacing traditionalmathematical operators with nonlinear operators which attempt tocharacterize the nonlinearity of image arithmetic. Table 1 summarizesand compares the LIP and PLIP model operators, where the specificinstance in which μ=M, γ=k=λ, and β=1, is of particular practicalinterest. Practically, for images in [0, M), the value of γ can eitherbe chosen such that γ≧M for positive γ or can take on any negativevalue.

TABLE 1 LIP and PLIP model mathematical operators LIP Model PLIP ModelGraytone g = M − I g = μ − I Addition${g_{1\mspace{14mu}}\mspace{14mu} g_{2}} = {g_{1} + g_{2} - \frac{g_{1}g_{2}}{M}}$${g_{1}\overset{\sim}{\oplus}g_{2}} = {g_{1} + g_{2} - \frac{g_{1}g_{2}}{\gamma}}$Subtraction${g_{1\mspace{14mu}}\mspace{14mu} g_{2}} = {M\frac{g_{1} - g_{2}}{M - g_{2}}}$${g_{1}\overset{\sim}{\Theta}g_{2}} = {k\frac{g_{1} - g_{2}}{k - g_{2}}}$Scalar Multiplication${c\mspace{14mu}\mspace{14mu} g_{1}} = {M - {M( {1 - \frac{g_{1}}{M}} )}^{c}}$${c\overset{\sim}{\otimes}g_{1}} = {{{\overset{\sim}{\varphi}}^{- 1}( {c{\overset{\sim}{\varphi}( g_{1} )}} )} = {\gamma - {\gamma\;( {1 - \frac{g_{1}}{\gamma}} )^{c}}}}$Isomorphic Transformation${\varphi(g)} = {{- M}{\mspace{11mu}\;}\ln\;( {1 - \frac{g}{M}} )}$${\overset{\sim}{\varphi}(g)} = {{- \lambda} \cdot {\ln^{\beta}( {1 - \frac{g}{\lambda}} )}}$${\varphi^{- 1}(g)} = {- {M\lbrack {1 - {\exp( {- \frac{g}{M}} )}} \rbrack}}$${{\overset{\sim}{\varphi}}^{- 1}(g)} = {\lambda\lbrack {1 - {\exp( \frac{- g}{\lambda} )}^{\frac{1}{\beta}}} \rbrack}$Graytone g₁ 

 g₂ = φ⁻¹(φ(g₁)φ(g₂)) g₁ {tilde over (·)} g₂ = {tilde over (φ)}⁻¹({tildeover (φ)}(g₁){tilde over (φ)}(g₂)) Multiplication Convolution w 

 g = φ⁻¹(w*φ(g)) w{tilde over (*)}g = {tilde over (φ)}⁻¹(w*{tilde over(φ)}(g))

When γ=256, the PLIP model operators revert to the LIP model operators.Furthermore, it can be shown that

$\begin{matrix}{{\lim\limits_{{\gamma }arrow\infty}\mspace{14mu}{\overset{\sim}{\varphi}(a)}} = {{\lim\limits_{{\gamma }arrow\infty}\mspace{14mu}{{\overset{\sim}{\varphi}}^{- 1}(a)}} = a}} & (3)\end{matrix}$

Since {tilde over (φ)} and {tilde over (φ)}⁻¹ are continuous functions,the PLIP model operators revert to arithmetic operators as |γ|approaches infinity and therefore, the PLIP model approaches standardlinear processing of graytone functions as |γ| approaches infinity.Thus, for the case of image fusion algorithms, an image algorithm whichutilizes standard linear processing operators can be found to be aninstance of an image algorithm using the PLIP model with γ=∞. Therefore,the PLIP framework can generalize any state-of-the-art fusion approachwhich has been developed or has yet to be developed. Image fusionalgorithms can be adapted using the PLIP model by providing amathematical formulation of multi-resolution decomposition schemes andfusion rules in terms of the model. This may be accomplished by directlyreplacing standard operators with PLIP operators, or by using theisomorphic transformation which defines the PLIP model. The graytone gof the input image I is first generated. By way of the isomorphictransformation, a multi-decomposition scheme at decomposition level l iscalculated by{tilde over (T)}({tilde over (y)} ₀ ^((l)))={tilde over (φ)}⁻¹(T({tildeover (φ)}({tilde over (y)} ₀ ^((l)))))  (4)where {tilde over (y)}₀ ⁽⁰⁾=g. Similarly, the inverse procedure beginsfrom transform coefficients at the highest decomposition level L. Eachsynthesis level reconstructs approximation coefficients at a scale l<Lby each synthesis level by{tilde over (T)} ⁻¹({tilde over (T)}({tilde over (y)} ₀ ^((l))))={tildeover (φ)}⁻¹({tilde over (φ)}({tilde over (T)}({tilde over (y)} ₀^((l))))))  (5)

Given {tilde over (y)}_(I) ₁ _(,0) ^((L)), {tilde over (y)}_(I) ₂ _(,0)^((L)), . . . {tilde over (y)}_(I) _(N−1) _(,0) ^((L)), {tilde over(y)}_(I) _(N) _(,0) ^((L)), the approximation coefficients of images I₁,I₂, . . . I_(N−1), I_(N) at the highest decomposition level L, yieldedusing a given parameterized logarithmic multi-resolution decompositiontechnique, the approximation coefficients for the fused image I′ at thehighest level of decomposition according to the PLIP model is given by{tilde over (y)} _(I′,0) ^((L))={tilde over (φ)}⁻¹(R _(D)({tilde over(φ)}({tilde over (y)} _(I) ₁ _(,0) ^((L)),{tilde over (φ)}({tilde over(y)} _(I) ₂ _(,0) ^((L)), . . . ,{tilde over (φ)}({tilde over (y)} _(I)_(N−1) _(,0) ^((L)),{tilde over (φ)}({tilde over (y)} _(I) _(N) _(,0)^((L)))))  (6)where R_(A) is an approximation coefficient fusion rule implementedusing standard arithmetic operators, respectively. Similarly, for eachof the i high-pass sub-bands of each of the N images at each level ofdecomposition l, the detail coefficient rule performed at each level ofdecomposition is given by{tilde over (y)} _(I′,i) ^((l))={tilde over (φ)}⁻¹(R _(D)({tilde over(φ)}({tilde over (y)} _(I) ₁ _(,i) ^((l))),{tilde over (φ)}({tilde over(y)} _(I) ₂ _(,i) ^((l))), . . . ,{tilde over (φ)}({tilde over (y)} _(I)_(N−1) _(,i) ^((l))),{tilde over (φ)}({tilde over (y)} _(I) _(N) _(,i)^((l)))))  (7)

FIGS. 7a-h illustrate the improvement which can be yielded in the fusionof multi-sensor data using the inventive class of PLIP image fusion, andthe necessity for the added model parameterization. The Q_(W) qualitymetric used for quantitatively assessing image fusion performanceimplies a better fusion for a higher value of Q_(W). The figure showsthat firstly, the PLIP model reverts to the LIP model with γ=M=256, andsecondly, the combination of source images using this extreme case maystill be visually unsatisfactory given the nature of the input images,even though the processing framework is based on a physically inspiredmodel.

FIGS. 7d-f illustrate the way in which fusion results are affected bythe parameterization, with the most improved fusion performance yieldedby the proposed approach using parameterized multi-resolutiondecomposition schemes and fusion rules relative to both the standardprocessing extreme and the LIP model extreme with γ=430.

FIGS. 7a,b are original “navigation” source images, image fusion resultsusing the LP/AM fusion rule, and PLIP model operators with (c) γ=256(LIP model case), Q_(W)=0.3467, (d) γ=300, Q_(W)=0.7802, (e) γ=430,Q_(W)=0.8200, (f) γ=700, Q_(W)=0.8128 (g) γ=10⁸, Q_(W)=0.7947, (h)standard mathematical operators, Q_(W)=0.7947

The described approaches, as described for grayscale images, can beextended for the case of color images, as shown in FIG. 8. In thisembodiment, the RGB color image is first converted 802 to a color spacesuch as HSV, XYZ, YCbCr, PCA, CIELAB, etc. The luminance channel 802 aresulting from the color space transformation is subjected to theinventive fusion processing 804 described above where secondary outputsusing different enhancement processes 804 b are generated and laterfused 804 c to produce the enhanced luminance channel 804 d of theimage. Depending on the nature of the color space, the other twochannels may be subjected to a color restoration procedure. Performingthe inverse color space transformation 808 with inputs from colorrestoration 806 yields an enhanced RGB output image I′.

In another aspect of the invention, in the context of noise removal fromimages, an iterative fusion-based technique for image denoising isprovided. One issue in image denoising is the tradeoff between effectivenoise removal and preservation of edges. In one embodiment, a methodfirst generates secondary outputs which differ by their degree ofsmoothing. The images I_(1,0) and I_(2,0) are the denoising resultsobtained using the least and most aggressive amounts of smoothing,respectively. The secondary outputs can be generated by any denoisingapproach, such as Gaussian filtering, anisotropic diffusion, totalvariation, wavelet approaches, etc. The salient features of I_(1,0),denoted by G, is also determined via an edge detector, statisticalfilter, or other means. At each iteration k, the aim is to improve edgepreservation of I_(2,k) while also effectively denoising the image. Thisis achieved by iteratively fusing information from I_(1,k) at edgelocations byI _(1,k+1) =G ^(α) I _(1,0)+(1−G ^(α))I _(2,k)  (8)I _(2,k+1) =G ^(α) I _(1,k+1)+(1−G ^(α))I _(2,k)  (9)where α is a parameter controlling the influence of G on the fusionprocess. It is seen that this procedure can effectively inject the edgeinformation from I_(1,k) into I_(2,k) while contributing substantiallyless noise. Thus, the inventive approach can be iterated K times, inwhich case the final denoising result is given by I_(2,K).

In another aspect of the invention, exemplary embodiments perform imageenhancement, image de-noising, and image fusion. Secondary outputs canbe fused together by exemplary embodiments of the invention. Fusionprocessing can be used both to generate secondary outputs, as well as tofuse secondary outputs to achieve the final output image. Thisprocessing make use of a novel set of HVS-inspired multi-scale toolsthat define multi-scale contrast coefficients in a way which isconsistent with the HVS, and alter multi-scale contrast coefficients forachieving various applications in signal and image processing.

The HVS perceives relative luminance changes for a large range ofbackground intensity values. Known as the luminance masking (LM)phenomena, the degree to which the HVS is sensitive to relative, and notabsolute, luminance differences varies with background illumination.Additionally, the HVS is sensitive not only to relative changes inluminance, but also to relative changes in contrast. This contrastmasking (CM) phenomena of HVS is one in which the visibility of acertain stimulus is reduced due to the presence of another one.Accordingly, we formulate HVS-inspired multi-scale transforms on the LMand CM phenomena of the HVS. In this case, the transforms are developedby directly emulating the HVS masking effects on transform domaincoefficients. It should be noted that any existing multi-scale signalrepresentation scheme can be used as the base transform for ourHVS-inspired multi-scale transform formulation and processing.

FIGS. 9a-c show a graphical depiction of the generation of (a)approximation, (b) detail and (c) HVS-inspired contrast coefficientsub-bands (absolute magnitudes) using the LP as the base transform.

Given the approximation and detail coefficients yielded using a standardmulti-scale transform, the inventive contrast measure first measures theLM contrast. The exact means by which this is accomplished is dependenton the given transform. In the LP domain, the LM contrast is given by

$\begin{matrix}{C_{LM}^{(n)} = \frac{y_{1}^{(n)}}{a_{1} + {{\overset{\_}{y}}_{0}^{({n + 1})}}^{\gamma_{1}}}} & (10)\end{matrix}$where a1 is a small constant, γ1 is a parameter which controls thedegree to which the luminance-masked contrast is affected by thebackground luminance, and x is an expansion function consisting of someup-sampling and interpolation operation. For the pyramidal-basedapproaches, the expansion is given byx =EXPAND(|x|)  (11)and for the wavelet-based approaches, it is given byx=ĝ _(R) *└ĝ _(C) *[|x|] _(↑2) _(R) ┘_(↑2) _(C)   (12)

The LM contrast is then masked with a local activity measure, which is afunction of the LM contrast, to yield the proposed luminance andcontrast masked (LCM) contrast. Again, the manner in which this isaccomplished depends on the base transform which is employed. In thecontext of the LP, the multi-scale LCM contrast is defined as:

$\begin{matrix}{C_{LCM}^{(n)} = \frac{C_{LC}^{(n)}}{a_{2} + {{\overset{\_}{C}}_{LC}^{({n + 1})}}^{\gamma_{2}}}} & (13)\end{matrix}$where a₂ is a small constant, and γ₂ is a parameter which controls thedegree to which the perceived contrast is affected by surroundingstimuli, as illustrated in FIGS. 9a-c . With a₁=a₂=γ₁=γ₂=0, the contrastreverts the detail coefficients given by standard multi-scaletransforms.

Other combinations incorporate the LM and CM characteristics of the HVSwith degrees dictated by γ₁ and γ₂. Thus, the contrast measuregeneralizes existing multi-scale transforms and algorithms using saidtransforms, in a manner which is motivated by known and relevant HVScharacteristics.

The standard transform coefficients can be recovered from themulti-scale LCM coefficients. For example in the LP domain, the LMcontrast can be calculated from the LCM contrast byC _(LM) ^((n)) =C _(LCM) ^((n)) ·[a ₂ +|C _(LC) ^((n+1))|^(γ) ² ]  (14)and the detail coefficients are calculated in terms of the LM contrastby (n)y ₁ ^((n)) =C _(LM) ^((n)) ·[a ₁ +| y ₀ ^((n+1))|^(γ) ¹ ]  (15)

The standard LP synthesis procedure can then be used to reconstruct theimage signal. As the inventive HVS masking is a completely invertibleprocedure, a novel class of multi-scale transforms is achievable, whichcombines HVS characteristics with commonly used multi-resolutiondecomposition schemes. Accordingly, the HVS-LP, HVS discrete wavelettransform (HVS-DWT), HVS stationary wavelet transform (HVS-SWT), and HVSdual-tree complex wavelet transform (HVS-DT-CWT) are provided. Theanalysis and synthesis steps required to calculate each of theseHVS-inspired transforms are given in Table 2. In each case, the analysisstage masks detail coefficients first by a measure of local luminanceand then by a measure of local activity to yield the HVS-inspiredcontrast. Each synthesis stage recovers the details coefficients fromthe HVS contrast coefficients.

TABLE 2 Analysis and synthesis stages of the proposed HVS-inspiredmulti-scale transforms HVS-LP HVS-DWT HVS-SWT HVS-DT-CWT Analysis LM$C_{LM}^{(n)} = \frac{y_{1}^{(n)}}{a_{1} + {{\overset{\_}{y}}_{0}^{({n + 1})}}^{\gamma_{1}}}$$C_{i,{LM}}^{(n)} = \frac{y_{i}^{(n)}}{a_{1} + {y_{0}^{(n)}}^{\gamma_{1}}}$$C_{i,{LM}}^{(n)} = \frac{y_{i}^{(n)}}{a_{1} + {y_{0}^{(n)}}^{\gamma_{1}}}$$C_{i,{LM}}^{(n)} = \frac{y_{i}^{(n)}}{a_{1} + {y_{0}^{({n + 1})}}^{\gamma_{1}}}$CM$C_{LCM}^{(n)} = \frac{C_{LM}^{(n)}}{a_{2} + {{\overset{\_}{C}}_{LM}^{({n + 1})}}^{\gamma_{2}}}$$C_{i,{LCM}}^{(n)} = \frac{C_{i,{LM}}^{(n)}}{a_{2} + {{\overset{\_}{C}}_{i,{LM}}^{({n + 1})}}^{\gamma_{2}}}$$C_{i,{LCM}}^{(n)} = \frac{C_{i,{LM}}^{(n)}}{a_{2} + {C_{i,{LM}}^{({n + 1})}}^{\gamma_{2}}}$$C_{i,{LCM}}^{(n)} = \frac{C_{i,{LM}}^{(n)}}{a_{2} + {{\overset{\_}{C}}_{i,{LM}}^{({n + 1})}}^{\gamma_{2}}}$Synthesis CM$C_{LM}^{(n)} = {C_{LCM}^{(n)} \cdot \lbrack {a_{2} + {{\overset{\_}{C}}_{LM}^{({n + 1})}}^{\gamma_{2}}} \rbrack}$$C_{i,{LM}}^{(n)} = {C_{i,{LCM}}^{(n)} \cdot \lbrack {a_{2} + {{\overset{\_}{C}}_{i,{LM}}^{({n + 1})}}^{\gamma_{2}}} \rbrack}$C_(i, LM)^((n)) = C_(i, LCM)^((n)) ⋅ [a₂ + C_(i, LM)^((n + 1))^(γ₂)]$C_{i,{LM}}^{(n)} = {C_{i,{LCM}}^{(n)} \cdot \lbrack {a_{2} + {{\overset{\_}{C}}_{i,{LM}}^{({n + 1})}}^{\gamma_{2}}} \rbrack}$LM$y_{1}^{(n)} = {C_{LM}^{(n)} \cdot \lbrack {a_{1} + {{\overset{\_}{y}}_{0}^{({n + 1})}}^{\gamma_{1}}} \rbrack}$y_(i)^((n)) = C_(i, LM)^((n)) ⋅ [a₁ + y₀^((n))^(γ₁)]y_(i)^((n)) = C_(i, LM)^((n)) ⋅ [a₁ + y₀^((n))^(γ₁)]y_(i)^((n)) = C_(i, LM)^((n)) ⋅ [a₁ + y₀^((n + 1))^(γ₁)]

FIG. 10 shows an exemplary system providing image enhancement processingbased on HVS multi-scale transforms. A forward HVS multi-scale transform1002 is applied to the original image. The brightness of the image isadjusted 1004 by altering the approximation coefficient sub-band at thehighest level of decomposition. Non-linear contrast mapping 1006 isapplied to the human visual contrast coefficient sub-bands at eachorientation and level of decomposition. In this stage, both contrastenhancement and dynamic range compression can be applied. Lastly, theinverse HVS transform 1008 is performed, yielding the enhanced image.

In addition to enhancing the contrast of certain image structures, animage enhancement process should also provide a (preferably direct)means of achieving dynamic range compression and global brightnessadjustments as deemed necessary for the given input image data. Theremay also be instances in which the necessities for an adequate imageenhancement are contradictory to the requirements dictated by a contrastenhancement framework, in which case these requirements must be relaxed.

The extension of existing direct multi-scale contrast enhancementframeworks to a more general direct multi-scale image enhancementframework demands that some additional requirements of contrastenhancement schemes be added. One such requirement is that the directenhancement procedure should yield a visually pleasing level ofbrightness. Thus, upon decomposing the input image into its multiplesub-bands, the brightness of the image is first adjusted. One of themost common means of brightness adjustment is by a power lawtransformation. However, power law transformations will performsimultaneous brightness adjustments, dynamic range compression, andthus, the degree to which they are tuned cannot be controlledindependently or directly. Equalization techniques have also beenconsidered. However, they may not accurately preserve edges as the localedge content is erratically treated. Given a suitable contrast measurewhich accurately encompasses elements of human vision, the brightness ofthe image can be sufficiently and accurately adjusted by adding aconstant to the approximation coefficient sub-band at the highest levelof decomposition. Again, exemplified in the case of the LP, thebrightness of the image is tuned byy′ ₀ ^((N)) =y ₀ ^((N)) +L  (16)where L is a brightness parameter. With L=0, the current brightnesslevel of the image is preserved.

The extension to a more general direct image enhancement framework alsodemands that some additional requirements mandated by contrastenhancement schemes be relaxed depending on the image context. This isto say that direct image enhancement procedures should provide a meansof directly achieving both contrast enhancement and dynamic rangecompression. For example, non-uniformities in lighting and shadows canbe perceived as large-scale contrast, and magnifying these contrasts mayonly exacerbate these negative effects. Thus, there may be instances inwhich overall visual quality is improved by compressing, or reducing thecontrast which is exhibited at a given scale. To this end, we relax therequirements of a non-linear mapping. Specifically, if the contrast at agiven scale is to be enhanced, areas of low contrast should be enhancedmore than areas of high contrast, and in this case the non-linearcontrast mapping function should not cause smoothing. However, if thecontrast is to be decreased, for example to remove non-uniformillumination, or to avoid signal clipping because caused by thebrightness adjustment step, areas of low contrast should have theircontrast decreased less than areas of high contrast, and the non-linearmapping function should not cause contrast enhancement. Accordingly, theHVS-inspired contrast non-linearly mapped byC′ _(LCM,i) ^((n))=sgn(C _(LCM,i) ^((n)))λ_(i) ^((n))(|C _(LCM,i)^((n))|)  (17)where the proposed non-linear contrast mapping function λ_(i) ^((n))(•)is

$\begin{matrix}{{\lambda_{i}^{(n)}(x)} = \{ \begin{matrix}{g_{1}^{(n)} \cdot x} & {x \leq T_{i}^{(n)}} \\{{g_{2}^{(n)} \cdot x} + {( {g_{1}^{(n)} - g_{2}^{(n)}} )T_{i}^{(n)}}} & {x > T_{i}^{(n)}}\end{matrix} } & (18)\end{matrix}$and g₁ ^((n))≧g₂ ^((n))≧0. This formulation allows (1) gain factors lessthan 1 to be considered for dynamic range compression, (2) theenhancement of high contrast areas to be tuned, and (3) an extension ofthe non-linear mapping to the DT-CWT coefficients in which phase ispreserved. Therefore, the inventive image enhancement allows for thebrightness and amount of contrast enhancement/dynamic range compressionto be controlled directly and independently of each other, maintainingthe spirit of a direct enhancement framework. To summarize the exactmeans by which direct image enhancement is achieved, the inventiveprocess, as formulated for an N level HVS-LP decomposition, is describedas follows:1) Generate an N+1 level Laplacian pyramid of I2) Measure the LM contrast of the original image3) Measure the LCM contrast of the original image4) Initialize y′₀ ^((N+1))=y₀ ^((N+1)), C′_(LM) ^((n))=C′_(LM) ^((n)).5) Adjust the brightness6) Calculate the enhanced LCM contrast by a non-linear mapping7) Calculate the enhanced LM contrast byC′ _(LM) ^((n)) =C′ _(LCM) ^((n)) ·[a ₂ +| C′ _(LM) ^((n+1))|^(γ) ² ]8) Calculate the enhanced detail coefficients byy′ ₁ ^((n)) =C′ _(LM) ^((n)) ·[a ₁ +| y′ ₀ ^((n+1))|^(γ) ¹ ]9) Calculate the enhanced approximation coefficients byy′ ₀ ^((n)) =y′ ₁ ^((n))+EXPAND[y′ ₀ ^((n+1))]10) The enhanced image I′=y′₀ ⁽⁰⁾

A similar formulation of the image enhancement algorithms is developedfor the other HVS multi-scale transforms whose analysis and synthesisstages were summarized in Table 2.

Due to the generalized formulation of the proposed approach, theprocessing encapsulates many existing multi-scale image enhancementapproaches. Table 3 summarizes direct enhancement processes which aregeneralized by the proposed approach. The proposed image enhancementprocessing not only uses a more comprehensive multi-scale model of theHVS, but also extends the use of non-linear detail coefficient mappingsto HVS-inspired contrast coefficients, and is capable of adjusting theoverall brightness of the enhanced result and achieving dynamic rangecompression.

TABLE 3 Comparison of existing direct enhancement algorithms withproposed method Dynamic Range Direct Coefficient Brightness Compres-Enhancement γ₁ γ₂ mapping Adjustment sion Multi-scale 0 0 Linear No NoUnsharp- Masking Laine's 0 0 Non-Linear No No Algorithm Tang's 1 0Linear No No Algorithm Inventive Vari- Vari- Non-linear Yes Yes methodable able

The inventive HVS-inspired multi-scale transforms are also useful forimage de-noising. Here, we exemplify the HVS-inspired multi-scaletransforms for image de-noising, and introduce a HVS-inspiredmulti-scale de-noising process based on the non-local means (NLM)principle, in which relevant patches within images are used asself-predictions for edge-preserving smoothing. The motivation for theinventive processing is to combine the state-of-the-art performance ofthe NLM algorithm with the advantages of multi-scale de-noisingapproaches, and additionally, multi-scale human visual processing. Theextension of the NLM algorithm to the multi-scale case can have variousrelated interpretations with some subtle differences. As with allmulti-scale de-noising approaches, the presented approach de-noises theedge structures occurring at different scales, as given by thecoefficient sub-bands at each level of decomposition. Moreover, it canalso be perceived as a extension of the NLM process in which multiplesimilarity windows are considered.

In practice, the NLM de-noising procedure is a spatial domain procedurewhich uses a weighted sum of pixels within a square search window W₁ ofthe central pixel to be de-noised. For each pixel of the noisy image,the weights in the local window are calculated by

$\begin{matrix}{{w_{k,l}( {k^{\prime},l^{\prime}} )} = {\frac{1}{Z( {k,l} )}{\exp\lbrack \frac{{{{I_{n}( \aleph_{k,l} )} - {I_{n}( \aleph_{k^{\prime},l^{\prime}} )}}}_{2,a}^{2}}{h^{2}} \rbrack}}} & (19) \\{{Z( {k,l} )} = {\sum\limits_{{({k^{\prime},l^{\prime}})} \in W_{1}}^{\;}{\exp\lbrack \frac{{{{I_{n}( \aleph_{k,l} )} - {I_{n}( \aleph_{k^{\prime},l^{\prime}} )}}}_{2,a}^{2}}{h^{2}} \rbrack}}} & (20)\end{matrix}$where h is a smoothing parameter. The term ∥I_(n)(

_(k,l))−I_(n)(

_(k′,l′))∥_(2,a) ² is the l₂ norm between the neighborhood centeredaround pixels (k,l) and (k′,l′),

_(k,l) and

_(k′,l′) respectively, which have been weighted using a Gaussian squareprofile W₂ of standard deviation a. At each pixel, image de-noising isachieved using the NLM process achieved by

$\begin{matrix}{{\hat{I}( {k,l} )} = {\sum\limits_{{({k^{\prime},l^{\prime}})} \in W_{1}}^{\;}{{w_{k,l}( {k^{\prime},l^{\prime}} )}{I_{n}( {k^{\prime},l^{\prime}} )}}}} & (21)\end{matrix}$

FIG. 11 shows a block diagram of an exemplary multi-scale NLM process1100 referred to as the HVS-NLM. Firstly, the image is decomposed 1102using any of the HVS-transforms described above, for example. Secondly,de-noising 1104 is applied at each level of decomposition. A NLMfiltering stage 1106 acts as an intermediary de-noising step. Namely,the approximation coefficients at each level of decomposition aredetermined, and the NLM procedure is applied. The intermediateapproximation coefficients at each level of decomposition are given byŷ _(0,int) ^((n))=NLM(y ₀ ^((n)) ,h ^((n)))  (22)where h^((n)) is the smoothing parameter at a given scale n. In order toperform level dependent smoothing in accordance to the intuition inwhich lower levels of decompositions are smoothed more aggressively thanhigher ones, the smoothing parameter at each scale is defined as

$\begin{matrix}{h^{(n)} = \frac{h^{(0)}}{\eta}} & (23)\end{matrix}$Where η is a parameter defining the relationship between smoothingfactors at each level of decomposition. The approximation coefficientsub-band at the highest level of decomposition 1108 can be defined aseitherŷ ₀ ^((N))=NLM(y ₀ ^((N)) ,h ^((N)))  (24)orŷ ₀ ^((N)) =y ₀ ^((N))  (25)

This is a subtle implementation option, in which the approximationsub-band at the highest level of decomposition can chosen to either bede-noised or not. The detail coefficients at each level of decompositionare determined by performing a single level analysis on eachintermediate approximation coefficient sub-band by(ŷ ₁ ^((n)) ,ŷ ₂ ^((n)) , . . . ,ŷ _(isb) ^((n)))=T(ŷ _(0,int)^((n−1)))  (26)

Because the processed detail coefficients at each level of decompositionas well as the approximation coefficients at the highest level ofdecomposition have now been determined, the transform space of thede-noised result is sufficiently filled. The de-noised image can becalculated by performing 1110 the inverse transformÎ=T ⁻¹(ŷ ₀ ^((N)) ,ŷ ₁ , . . . ,ŷ _(isb))  (27)

In another aspect of the invention, a set of image fusion processesbased on the HVS-inspired multi-scale tools are provided with inventivefusion of perceptually-based contrast coefficients provided by thesetransforms and a novel means of fusing approximation coefficients basedon an adaptive similarity-based weighting scheme. Accordingly, theproposed approximation coefficient rule is developed, using the HVS-LPfor this specific formulation. The proposed approximation coefficientrule begins by first computing an estimate of the fused approximationcoefficients, given by the uniform average

$\begin{matrix}{y_{i_{F,0}}^{(N)} = \frac{y_{I_{1},0}^{(N)} + y_{I_{2},0}^{(N)}}{2}} & (28)\end{matrix}$

When using a global, uniform averaging scheme, the locally lesspertinent information will tend to “wash-out” the locally more pertinentinformation. However, some inferences can still be made from thisinitial uniform fusion. Namely, one would observe that such a fusionwould still be more perceptually similar to the stimulus than to theuniform background. Thus, using the proposed weighting scheme, thestimulus would be given higher weight than the uniform background, in away which was not directly related to measuring the amount of salientfeatures in each of the source images. Therefore, this hypotheticallydemonstrates that the degree to which the initial estimate isperceptually similar to each source image could feasibly used toadaptively determine the sensor weights for the fusion procedure. Adifferent initial estimate could have also been used. The effectivenessof such a weighting scheme is thus dependent on the establishment of aperceptually-driven similarity metric, such as the gradient structuralsimilarity index (GSSIM). Accordingly, the fusion weights are defined by

$\begin{matrix}{{{w_{1}( {i,j} )} = \frac{{{{GSSIM}_{y_{i_{F,0}^{(N)},y_{I_{1},0}^{(N)}}}( {i,j} )}}^{\alpha}}{{{{GSSIM}_{y_{i_{F,0}^{(N)},y_{I_{1},0}^{(N)}}}( {i,j} )}}^{\alpha} + {{{GSSIM}_{y_{i_{F,0}^{(N)},y_{I_{2},0}^{(N)}}}( {i,j} )}}^{\alpha}}}{{w_{2}( {i,j} )} = {1 - {w_{1}( {i,j} )}}}} & (29)\end{matrix}$where α is an empirical, user-defined parameter, which has been added todictate how much stronger the locally more similar source image shouldbe weighted. The weighted fusion is then given byy _(I) _(F) _(,0) ^((N))(i,j)=w ₁(i,j)y _(I) ₁ _(,0) ^((N))(i,j)+w₂(i,j)y _(I) ₂ _(,0) ^((N))(i,j)  (30)

Practically, the α parameter dictates the degree to which the weightscan feasibly deviate from those in the uniform averaging case, with thisdeviation derived from the local similarity between the initial fusedimage and each of the source images. If α=0, then w₁(i,j)=w₂(i,j)=0.5 atall pixel locations, and thus, the proposed method reverts to theuniform averaging case. For non-negative values of α, the fusion weightsare adaptively altered according to the local similarity assessmentwhich has been described, with the degree of the alteration dictated byα. This is to say that the adaptive weighting scheme can determine whichsensor should have a greater weight, and the α parameter dictatesexactly how much more of a greater weight it should have. The inclusionof this α parameter generalizes the approximation coefficient fusionrule between uniform weighting (α=0), simple selection (α=∞), and valuesin between this range. In practice, it is sufficient to set α=1, as aformidable improvement in the fusion quality is already yielded in thiscase.

The inventive process also determines the contrast coefficients of thefused image C_(I) _(F) _(,LCM) ^((n)) at each scale n using an absolutecontrast coefficient maximum rule

$\begin{matrix}{{C_{I_{F},{LCM}}^{(n)}( {i,j} )} = \{ \begin{matrix}{C_{I_{1},{LCM}}^{(n)}( {i,j} )} & {{{C_{I_{1},{LCM}}^{(n)}( {i,j} )}} > {{C_{I_{2},{LCM}}^{(n)}( {i,j} )}}} \\{C_{I_{2},{LCM}}^{(n)}( {i,j} )} & {{{C_{I_{1},{LCM}}^{(n)}( {i,j} )}} \leq {{C_{I_{2},{LCM}}^{(n)}( {i,j} )}}}\end{matrix} } & (31)\end{matrix}$

The detail coefficients of the fused image at each scale are thencomputed by using the synthesis equations of the HVS-LP.

The performance of fusion processes deteriorate considerably when theinput source images themselves are corrupted by noise. One solution forremedying this is to de-noise the images as a pre-processing step, or“denoise-then-fuse.” Alternatively, another solution is to fuse thenoisy images and de-noise this result, or “fuse-then-denoise.” Asdescribed, these processes are referred to as separate-fusion-de-noising(SFD) processes, because the fusion and de-noising procedures areperformed independently of each other. In contrast, in ajoint-fusion-de-noising (JFD) architecture, fusion and de-noising are ina sense performed simultaneously. Specifically, the input images areagain fused, but in contrast to the SFD approach, one image may alsohelp to de-noise another more effectively.

The extension of the NLM framework for joint de-noising and image fusionis motivated by FIGS. 12a-c , which show (a) image fusion, (b) imagede-noising, and (c) JFD visualized as averaging along different axes.Visualizing the input images as matrices placed on a vertical stack,image fusion can be interpreted as an averaging procedure performedvertically, as in FIG. 12a , while image de-noising, horizontally, as inFIG. 12b . In fact, the de-noising operation can itself be viewed as avertical averaging procedure, but of shifted and adaptively weightedversions of the image(s). Thus, the JFD procedure can be one in whichboth vertical and horizontal averaging are performed simultaneously.

FIG. 13a shows a separate-fusion-de-noising (SFD) architecture and FIG.13b shows a joint-fusion-de-noising (JFD) architecture. An inventiveNLM-based JFD procedure is now formulated, which is easily applicableboth to the NLM and HVS-NLM procedures. For ease of clarity, theprocedure will be defined in the context of NLM. The extension to theHVS-NLM case is achieved by performing the JFD scheme at each level ofdecomposition, in a manner consistent to that which has been previouslydescribed.

The images I₁ and I₂ are fused in some way, yielding the noisy fusionresult I_(F). Here, they are fused using the proposed adaptiveHVS-inspired approach. Three weights of the form w_(k,l) ^(q,r) aredefined, where q is the reference image (in this case, the fused imageI_(F)) and r is the template image. The weights are given by

$\begin{matrix}{{w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )} = {\exp\lbrack \frac{{{{I_{F}( \aleph_{k,l} )} - {I_{F}( \aleph_{k^{\prime},l^{\prime}} )}}}_{2,a}^{2}}{h^{2}} \rbrack}} & (32) \\{{w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )} = {\exp\lbrack \frac{{{{I_{F}( \aleph_{k,l} )} - {I_{1}( \aleph_{k^{\prime},l^{\prime}} )}}}_{2,a}^{2}}{h^{2}} \rbrack}} & (33) \\{{w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )} = {\exp\lbrack \frac{{{{I_{F}( \aleph_{k,l} )} - {I_{2}( \aleph_{k^{\prime},l^{\prime}} )}}}_{2,a}^{2}}{h^{2}} \rbrack}} & (34)\end{matrix}$

To determine if templates from I_(F), I₁, or I₂ should be used tode-noise I_(F), a hard decision rule is defined as

$\begin{matrix}{{\lambda_{k,l}^{F}( {k^{\prime},l^{\prime}} )} = \{ \begin{matrix}1 & {{{w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )} \geq {P \cdot {w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )}}},{{w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )} \geq {P \cdot {w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )}}}} \\0 & {otherwise}\end{matrix} } & (35) \\{{\lambda_{k,l}^{1}( {k^{\prime},l^{\prime}} )} = \{ \begin{matrix}1 & {{{P \cdot {w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )}} \geq {w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )}},{{w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )} \geq {w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )}}} \\0 & {otherwise}\end{matrix} } & (36) \\{{\lambda_{k,l}^{2}( {k^{\prime},l^{\prime}} )} = \{ \begin{matrix}1 & {{{P \cdot {w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )}} \geq {w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )}},{{w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )} \geq {w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )}}} \\0 & {otherwise}\end{matrix} } & (37)\end{matrix}$where P is a penalty parameter which for practical purposes can be setto 1. Accordingly, the fused and de-noised result is given by

                                           (38)${{\hat{I}}_{F}( {k,l} )} = \frac{\begin{matrix}{{\sum\limits_{j \in W}^{\;}\;{{\lambda_{k,l}^{F}( {k^{\prime},l^{\prime}} )}{w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )}I_{F}( {k,l} )}} + {\sum\limits_{j \in W}^{\;}{{\lambda_{k,l}^{1}( {k^{\prime},l^{\prime}} )}{w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )}I_{F}( {k,l} )}} +} \\{\sum\limits_{j \in W}^{\;}{{\lambda_{k,l}^{2}( {k^{\prime},l^{\prime}} )}{w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )}I_{F}( {k,l} )}}\end{matrix}}{\begin{matrix}{{\sum\limits_{j \in W}^{\;}{{\lambda_{k,l}^{F}( {k^{\prime},l^{\prime}} )}{w_{k,l}^{F,F}( {k^{\prime},l^{\prime}} )}}} + {\sum\limits_{j \in W}^{\;}{{\lambda_{k,l}^{1}( {k^{\prime},l^{\prime}} )}{w_{k,l}^{F,1}( {k^{\prime},l^{\prime}} )}}} +} \\{\sum\limits_{j \in W}^{\;}{{\lambda_{k,l}^{2}( {k^{\prime},l^{\prime}} )}{w_{k,l}^{F,2}( {k^{\prime},l^{\prime}} )}}}\end{matrix}\quad}$

If I₁=I₂, then I₁=I₂=I_(F), the JFD process reverts to the de-noisingprocedure. If h=0, no de-noising is performed, the JFD process revertsto the fusion procedure. If P=0, then the process becomes a SFDprocedure. In its most general sense, this process simultaneouslyperforms image fusion and de-noising, as the source images I₁ and I₂ areused both to generate and de-noise the fused image.

It is understood that image fusion has been used extensively to fusemultiple source images obtained from multiple different sensors orcapture techniques. However, there has been no systematic, generalizedapplication of image fusion developed for the case in which only asingle input image is available. This is generally expected to be thecase for many image processing applications, such as image enhancement,edge detection and image denoising. Exemplary embodiments of theinvention combine the outputs of existing image processing to yield theoutput of the system. Thus, by using image fusion concepts, certainembodiments of the present invention attempt to improve the quality ofexisting image processing by combining the respective advantages of eachtechnique. Additionally, processes for image enhancement, denoising, andfusion are developed based on human visual system-inspired processingtechniques. These inventive processes outperform standard imageprocessing techniques because of their generalized multi-scale andperceptually-based formulations.

One issue in image enhancement is the different and often timescontradictory criteria on which image enhancement algorithms have beenformulated. Indirect image enhancement algorithms provide visuallypleasing global enhancement, but can degrade the integrity of local edgefeatures. Conversely, direct enhancements can provide adaptive edgeenhancement, but generally do not provide satisfactory globalenhancement. Thus, it is advantageous to combine the outputs of thesetwo types of enhancements algorithms to leverage on the benefits ofeach. Moreover, given the characteristics of the direct and indirectimage enhancements themselves, it is possible to more effectivelycombine their outputs. Using image fusion methodologies, exemplaryembodiments of the present invention provide global enhancement whileretaining the contrast information of the original image. Therefore,illustrative embodiments of the present invention are able toeffectively decouple edge enhancement and global enhancement proceduresso that they can be tuned separately. Doing so allows the global andedge enhancement procedures to be performed in series with reducedartifacting, and significantly improves the visual quality ofenhancement results.

Image denoising has been a continuously studied problem in imageprocessing due to the fundamental tradeoff between effective noiseremoval and accurate preservation of edges. In general, the greater theamount of smoothing which is applied to the noisy observation, the morethe noise is suppressed and edge details and textures eliminated.Exemplary embodiments of the present invention combine the outputs ofdifferent denoising processes and/or parameter sets to yield the finaldenoised output. Specifically, an iterative fusion-based approach isused to fuse images smoothed to varying degrees to more accuratelyretain edges while also removing noise. Experimental results show thatthe proposed approach can improve image denoising results qualitativelyand quantitatively using many secondary denoising processes.

Exemplary embodiments of the invention provide image fusion integratedto improve existing image processing. In general, the image enhancementprocessing described above can be classified as indirect or direct imageenhancement based on the criteria for which enhancement is achieved.Known image processing techniques do not provide a means of using imagefusion for image enhancement, image denoising, or any application, whereonly a single input source image is available. By employing PLIP modelarithmetic operators can tune the image fusion process according toimage-dependent characteristics. Standard mathematical operators may beviewed as an instance of the PLIP model arithmetic, and therefore, theimage fusion adopting the PLIP model can be viewed as a generalizationof existing image fusion using standard mathematical operators. Theapplication of image fusion in this manner poses a means of combiningthe advantages of the numerous image enhancement methods that have beendeveloped according to their various criteria. In one embodiment, theproposed system provides a means of combining image enhancementprocesses to improve both the local and global image contrast. Anotherembodiment decouples the global and adaptive enhancement procedures sothat they can be tuned independently and be used in series.

It is understood that image enhancement processing described herein isdesirable in a wide range of applications, such as object detection,object recognition, and other computer vision and computer-aideddecision systems, security, medical, digital image retouching softwarepackages, and the like.

FIG. 15 shows an exemplary computer 800 that can perform at least partof the processing described herein. The computer 1500 includes aprocessor 1502, a volatile memory 1504, a non-volatile memory 1506(e.g., hard disk), an output device 1507 and a graphical user interface(GUI) 1508 (e.g., a mouse, a keyboard, a display, for example). Thenon-volatile memory 1506 stores computer instructions 1512, an operatingsystem 1516 and data 1518. In one example, the computer instructions1512 are executed by the processor 1502 out of volatile memory 1504. Inone embodiment, an article 1520 comprises non-transitorycomputer-readable instructions.

Processing may be implemented in hardware, software, or a combination ofthe two. Processing may be implemented in computer programs executed onprogrammable computers/machines that each includes a processor, astorage medium or other article of manufacture that is readable by theprocessor (including volatile and non-volatile memory and/or storageelements), at least one input device, and one or more output devices.Program code may be applied to data entered using an input device toperform processing and to generate output information.

The system can perform processing, at least in part, via a computerprogram product, (e.g., in a machine-readable storage device), forexecution by, or to control the operation of, data processing apparatus(e.g., a programmable processor, a computer, or multiple computers).Each such program may be implemented in a high level procedural orobject-oriented programming language to communicate with a computersystem. However, the programs may be implemented in assembly or machinelanguage. The language may be a compiled or an interpreted language andit may be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program may be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network. Acomputer program may be stored on a storage medium or device (e.g.,CD-ROM, hard disk, or magnetic diskette) that is readable by a generalor special purpose programmable computer for configuring and operatingthe computer when the storage medium or device is read by the computer.Processing may also be implemented as a machine-readable storage medium,configured with a computer program, where upon execution, instructionsin the computer program cause the computer to operate.

Processing may be performed by one or more programmable processorsexecuting one or more computer programs to perform the functions of thesystem. All or part of the system may be implemented as, special purposelogic circuitry (e.g., an FPGA (field programmable gate array) and/or anASIC (application-specific integrated circuit)).

Having described exemplary embodiments of the invention, it will nowbecome apparent to one of ordinary skill in the art that otherembodiments incorporating their concepts may also be used. Theembodiments contained herein should not be limited to disclosedembodiments but rather should be limited only by the spirit and scope ofthe appended claims. All publications and references cited herein areexpressly incorporated herein by reference in their entirety.

What is claimed is:
 1. A method for performing image fusion for imageprocessing, comprising: receiving a first input image (I); generating,using a computer processor, N secondary outputs (I_(n), n=1, 2, . . .N−1, N) using N combinations of secondary image processes and/orparameter sets derived from the first input image (I); and fusing theintermediate outputs to yield a processed output (I′) includingdecomposing the first input image into different grayscalerepresentations, separately processing the different grayscalerepresentation to provide outputs, and fusing the outputs to yield theprocessed output (I′).
 2. The method according to claim 1, wherein thefirst input image was captured by a single sensor.
 3. The methodaccording to claim 1, wherein the first input image data was capturedvia multiple sensors.
 4. The method according to claim 1, furtherincluding fusing the intermediate outputs based on at least oneparameter corresponding to one or more of image enhancement, imagedenoising, edge detection, object detection, object recognition, and/orcomputer aided-decision making.
 5. The method according to claim 1,further including decomposing the first input image into differentsubimages and fusing the subimages by different processes.